{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 作业3：前馈神经网络+文生图模型动手实践"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 数值稳定的算法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在编写激活函数或计算损失函数时，经常会遇到一些极端的取值，如果不对其进行适当的处理，很可能导致计算结果出现 `NaN` 或其他异常结果，影响程序的正常运行。本题将着重练习若干数值稳定的计算方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第1题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(a) 考虑 Sigmoid 函数 $$\\sigma(x)=\\frac{e^x}{1+e^x}$$\n",
    "\n",
    "请利用 PyTorch 编写一个函数 `sigmoid(x)`，令其可以接收一个 Tensor `x`，返回 Sigmoid 函数在 `x` 上的取值。不可直接调用 `torch.sigmoid()`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "\n",
    "def sigmoid(x):\n",
    "    # 完成函数体\n",
    "    e = torch.exp(-torch.abs(x))\n",
    "    numer = torch.where(x>=0, 1.0, e)\n",
    "    denom = 1.0 + e\n",
    "    return numer / denom"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一个简单的测试："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.0000e+00, 0.0000e+00, 4.5398e-05, 5.0000e-01, 9.9995e-01, 1.0000e+00,\n",
      "        1.0000e+00])\n",
      "tensor([0.0000e+00, 3.7835e-44, 4.5398e-05, 5.0000e-01, 9.9995e-01, 1.0000e+00,\n",
      "        1.0000e+00])\n"
     ]
    }
   ],
   "source": [
    "x = torch.tensor([-1000.0, -100.0, -10.0, 0.0, 10.0, 100.0, 1000.0])\n",
    "\n",
    "# PyTorch 自带函数\n",
    "print(torch.sigmoid(x))\n",
    "\n",
    "# 上面编写的函数\n",
    "print(sigmoid(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(b) 如果出现异常取值，思考可能的原因是什么。（提示：Sigmoid 函数真实的取值范围是多少？分子和分母的取值范围又是什么？是否可以对 Sigmoid 函数的表达式进行某种等价变换？）请再次尝试编写 Sigmoid 函数。如果一切正常，可忽略此问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在此处插入代码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第2题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(a) 考虑 Tanh 函数 $$\\sigma(x)=\\frac{e^x-e^{-x}}{e^x+e^{-x}}$$\n",
    "\n",
    "请利用 PyTorch 编写一个函数 `tanh(x)`，令其可以接收一个 Tensor `x`，返回 Tanh 函数在 `x` 上的取值。不可直接调用 `torch.tanh()`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "\n",
    "def tanh(x):\n",
    "    # 完成函数体、\n",
    "    e = torch.exp(-torch.abs(2*x))\n",
    "    numer = torch.where(x>=0, 1.0 - e, e - 1.0)\n",
    "    denom = 1.0 + e\n",
    "    return numer / denom"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一个简单的测试："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-1., -1., -1.,  0.,  1.,  1.,  1.])\n",
      "tensor([-1., -1., -1.,  0.,  1.,  1.,  1.])\n"
     ]
    }
   ],
   "source": [
    "x = torch.tensor([-1000.0, -100.0, -10.0, 0.0, 10.0, 100.0, 1000.0])\n",
    "\n",
    "# PyTorch 自带函数\n",
    "print(torch.tanh(x))\n",
    "\n",
    "# 上面编写的函数\n",
    "print(tanh(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(b) 如果出现异常取值，思考可能的原因是什么。请再次尝试编写 Tanh 函数。如果一切正常，可忽略此问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在此处插入代码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第3题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(a) 考虑 Softplus 函数 $$\\mathrm{softplus}(x)=\\log(1+e^x)$$\n",
    "\n",
    "请利用 PyTorch 编写一个函数 `softplus(x)`，令其可以接收一个 Tensor `x`，返回 Softplus 函数在 `x` 上的取值。不可直接调用 `torch.nn.functional.softplus()`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "def softplus(x):\n",
    "    # 完成函数体\n",
    "    e = torch.exp(-torch.abs(x)) # e^(-x)\n",
    "    log1e = torch.log(1 + e) # log(1+e)\n",
    "    s_x = torch.where(x >= 0, x + log1e, log1e) # s(x) = log(1+e^x)\n",
    "    return s_x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一个简单的测试："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.0000e+00, 3.7835e-44, 4.5399e-05, 6.9315e-01, 1.0000e+01, 1.0000e+02,\n",
      "        1.0000e+03])\n",
      "tensor([0.0000e+00, 0.0000e+00, 4.5418e-05, 6.9315e-01, 1.0000e+01, 1.0000e+02,\n",
      "        1.0000e+03])\n"
     ]
    }
   ],
   "source": [
    "x = torch.tensor([-1000.0, -100.0, -10.0, 0.0, 10.0, 100.0, 1000.0])\n",
    "\n",
    "# PyTorch 自带函数\n",
    "print(torch.nn.functional.softplus(x))\n",
    "\n",
    "# 上面编写的函数\n",
    "print(softplus(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(b) 如果出现异常取值，思考可能的原因是什么。请再次尝试编写 Softplus 函数。如果一切正常，可忽略此问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在此处插入代码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第4题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在作业2第2题中，为了计算损失函数，我们先计算了 $\\hat{\\rho}=\\mathrm{sigmoid}(X\\beta)$，然后再与 $y$ 计算 $l(y,\\hat{\\rho})=-y\\log \\hat{\\rho} - (1-y) \\cdot \\log(1-\\hat{\\rho})$。但当 $\\hat{\\rho}$ 非常接近0或1时，可能就会出现 $\\log(0)$ 错误。根据本次作业第1题和第3题的结果，请思考是否有更稳定的数值算法，并重新编写损失函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_fn_logistic(bhat, x, y):\n",
    "    # 编写函数主体，替换这里的代码\n",
    "    rho_x = torch.matmul(x, bhat) # rho(x) = XB\n",
    "    s_x = softplus(rho_x)\n",
    "    loss = -torch.mean(y * (rho_x - s_x) + (1 - y) * (-s_x))\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 前馈神经网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第5题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "利用模块化编程（参考课件 `lec5-fnn.ipynb` 中的实现），在如下模拟数据上构建一个 Logistic 回归模型（包含截距项），并利用自动微分和梯度下降法求解回归系数。要求使用 PyTorch 中的 `nn.Linear` 模块完成模型构建。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "tuple index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[10], line 2\u001b[0m\n\u001b[0;32m      1\u001b[0m n \u001b[38;5;241m=\u001b[39m x\u001b[38;5;241m.\u001b[39mshape[\u001b[38;5;241m0\u001b[39m]  \u001b[38;5;66;03m# 样本量\u001b[39;00m\n\u001b[1;32m----> 2\u001b[0m p \u001b[38;5;241m=\u001b[39m x\u001b[38;5;241m.\u001b[39mshape[\u001b[38;5;241m1\u001b[39m]  \u001b[38;5;66;03m# 输入维度\u001b[39;00m\n\u001b[0;32m      3\u001b[0m d \u001b[38;5;241m=\u001b[39m y\u001b[38;5;241m.\u001b[39mshape[\u001b[38;5;241m1\u001b[39m]  \u001b[38;5;66;03m# 输出维度\u001b[39;00m\n\u001b[0;32m      4\u001b[0m r \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m5\u001b[39m           \u001b[38;5;66;03m# 隐藏层维度\u001b[39;00m\n",
      "\u001b[1;31mIndexError\u001b[0m: tuple index out of range"
     ]
    }
   ],
   "source": [
    "n = x.shape[0]  # 样本量\n",
    "p = x.shape[1]  # 输入维度\n",
    "d = y.shape[1]  # 输出维度\n",
    "r = 5           # 隐藏层维度\n",
    "import torch.nn as nn\n",
    "\n",
    "class MyModel(nn.Module):\n",
    "    def __init__(self, input_dim, hidden_dim, output_dim):\n",
    "        super(MyModel, self).__init__()\n",
    "        self.w1 = nn.Parameter(torch.randn(input_dim, hidden_dim))\n",
    "        self.b1 = nn.Parameter(torch.rand(hidden_dim, 1))\n",
    "        self.w2 = nn.Parameter(torch.randn(hidden_dim, output_dim))\n",
    "        self.b2 = nn.Parameter(torch.rand(output_dim, 1))\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 完成此处程序\n",
    "        z1 = torch.matmul(x, self.w1) + self.b1.reshape(1, r)\n",
    "        a1 = softplus(z1)\n",
    "        z2 = torch.matmul(a1, self.w2) + self.b2\n",
    "        a2 = sigmoid(z2)\n",
    "        return a2\n",
    "\n",
    "np.random.seed(123456)\n",
    "torch.random.manual_seed(123456)\n",
    "\n",
    "r = 3\n",
    "model = MyModel(input_dim=p, hidden_dim=r, output_dim=d)\n",
    "print(list(model.parameters()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "\n",
    "dat = pd.DataFrame(x.numpy(), columns=[\"x1\", \"x2\"])\n",
    "dat[\"y\"] = y.numpy()\n",
    "sns.scatterplot(data=dat, x=\"x1\", y=\"x2\", hue=\"y\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在此处完成代码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "完成模型训练后，利用得到的模型对如下测试集数据进行预测（概率 >0.5 判为1，反之判为0），计算分类的正确率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(654321)\n",
    "torch.random.manual_seed(654321)\n",
    "\n",
    "ntest = 200\n",
    "xtest, ytest = gen_data(ntest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在此处完成代码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第6题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "修改第5题中的线性模型，将其变为一个两层的前馈神经网络，隐藏神经元数量为32，使用 ReLU 激活函数。然后重新训练模型（可尝试使用不同的学习率和迭代次数），并对测试集进行预测，计算分类的正确率（目标是 >90%）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在此处完成代码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 动手实践项目：基于上财 AI 平台的文生图模型搭建"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "观看视频 [https://www.bilibili.com/video/BV19qmgY5ED5](https://www.bilibili.com/video/BV19qmgY5ED5)，学习其中的方法，在自己的账号中搭建一个文生图模型应用。发挥你的创意，利用搭建好的模型生成若干图片，并将操作界面的截图插入下方。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
